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ABSTRACT 

The  test  and  evaluation  of  complex  socio-technical  systems  is  difficult.  This  is 
especially  so  when  the  test  and  evaluation  (T&E)  community  involved  tends  to 
associate  itself  closely  with  the  physical  sciences.  The  behavioural  sciences  have 
developed  methods  to  address  such  problems  and  have  shown  that  these  methods  are 
capable  of  providing  rigorous  results.  A  discussion  of  the  worth  of  qualitative  data 
and  an  overview  of  the  behavioural  science  methods  likely  to  be  appropriate  for 
evaluating  complex  systems  involving  humans  is  provided.  A  minor  case  study 
illustrating  the  application  of  qualitative  data  analysis  methods  to  the  evaluation  of  the 
Australian  Army's  Battlefield  Command  Support  System  (BCSS)  is  discussed  in  order 
to  illustrate  the  merit  of  behavioural  science  methods  for  T&E. 
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Applying  Behavioural  Science  Techniques  to  the  Test 
and  Evaluation  of  Complex  Systems 


Executive  Summary 

This  report  is  adapted  from  a  conference  paper  submitted  for  presentation  at  the 
November  2000  Systems  Engineering  and  Test  &  Evaluation  Conference  (SETE  2000)  in 
Brisbane,  Australia.  The  report  points  out  that  while  systems  engineering  (SE)  is 
concerned  with  problems  ranging  from  the  highly  abstract  to  the  highly  practical,  Test 
and  Evaluation  (T&E)  tends  to  concentrate  on  the  practical  problems.  The  behavioural 
sciences  offer  an  opportunity  for  T&E  practitioners  to  use  subjective  as  well  as 
qualitative  data  and  analysis  techniques  for  the  more  complex  socio- technical  systems. 
The  application  of  such  techniques  to  the  evaluation  of  components  of  the  Australian 
Army  Battlefield  Command  Support  System  is  discussed  as  a  minor  case  study. 
Systems  Engineering  theory  acknowledges  the  impact  and  influence  that  the  human 
element  has  on  the  effectiveness  of  systems.  The  T&E  community  needs  to  acquire  the 
skills  to  assess  this  impact.  A  T&E  community  with  a  broader  level  of  expertise  will  be 
better  able  to  evaluate  new  acquisitions  of  military  equipment. 
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1.  Introduction 


There  are  many  definitions  of  'system'  in  the  English  language  but  the  concept  of 
'system'  used  in  this  report  is  that  expressed  in  General  Systems  Theory  and  reflected 
in  the  emerging  discipline  of  systems  engineering.  A  system  then  is  to  be  considered 
as  a  set  of  components  with  interrelationships  such  that  they  form  a  unity.  They  have 
the  properties  of  emergence,  heirarchy,  communications,  and  control  (Checkland, 
1981). 

Jackson  and  Keys  (1984,  p.  475)  make  a  distinction  between  complex  and  simple 
systems.  They  note  the  small  size  of  the  components  and  the  regular  interaction 
between  the  components  for  simple  systems  as  opposed  to  the  large  number  of 
components  and  high  interrelationships  of  complex  systems.  Checkland  (1981) 
describes  systems  as  falling  into  the  four  classes  of  natural,  designed,  designed 
abstract,  or  human  activity  systems.  A  term  often  used,  in  a  similar  fashion  to 
Checkland's  human  activity  system,  is  'socio-technical  systems'.  Kline  (1995,  p.  60) 
illustrates  this  when  he  defines  them  as  linking  humans  with  hardware  or  tools  so  as  to 
perform  tasks  that  people  want  done.  Socio-technical  or  human  activity  systems 
generally  fall  into  the  category  of  complex  systems,  while  designed  systems  are  usually 
considered  not  to  be  complex. 

The  Test  and  Evaluation  (T&E)  of  designed  systems  is  relatively  easier  than  that  of 
complex  systems.  With  designed  systems,  there  is  a  structured  problem  to  be  solved 
and  it  can  be  taken  that  the  humans  involved  are  united  in  their  views  of  the  nature  of 
the  problem  and  its  eventual  solution.  T&E  has  traditionally  been  more  involved  with 
designed  than  with  complex  systems,  with  a  resulting  close  association  with  the 
various  physical  sciences  such  as  physics  and  chemistry,  as  well  as  mathematics.  This 
is  understandable  as  the  driving  force  for  T&E  to  date  has  principally  come  from 
industries  such  as  defence  and  aerospace.  Almost  always,  the  goals  are  clear,  are 
agreed  by  most  if  not  all  those  involved,  there  is  an  emphasis  on  physical  entities,  and 
there  is  a  total  dependence  on  technology  to  achieve  these  goals.  T&E  has  become  the 
domain  of  the  scientist  and  the  engineer  who  seek  quantitative  data  in  order  to  test  and 
evaluate  technological  solutions  to  problems.  So  long  as  there  is  something  that  can  be 
weighed  or  whose  temperature  can  be  assessed  or  that  can  be  touched  or  felt,  or  such¬ 
like,  then  there  is  a  way  of  including  it  in  a  T&E  programme. 

Socio-technical  or  human  activity  systems  problems  are  generally  unstructured  and 
there  is  more  likely  to  be  a  plurality  of  views  on  the  nature  of  a  successful  solution. 
The  effectiveness  of  a  public  health  programme  or  of  a  mass  transit  system  will  differ 
dependent  on  the  viewpoint  of  those  concerned.  The  viewpoint  of  the  commuter  using 
a  mass  transit  system  desiring  feelings  of  comfort,  service,  and  security  may  well  differ 
from  that  of  the  politician  seeking  to  allocate  funds  between  several  programmes. 
Often  there  are  no  numbers  but  there  is  a  lot  of  narrative  or  words  ie.,  there  is  little 
quantitative  data  but  plenty  of  qualitative  data.  It  would  seem  that  T&E  practitioners 
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are  at  a  loss  to  develop  a  T&E  programme  when  faced  with  such  complex  system 
problems  as  these,  because  all  their  experience  is  with  less  complex  designed  systems. 
This  makes  it  difficult  when  faced  with  the  T&E  of  systems  in  the  defence  and 
aerospace  areas  that  should  be  addressed  as  complex  systems,  such  as  those  associated 
with  the  Command  and  Control  (C2)  of  military  operations.  C2  comprises  not  only  an 
interaction  between  pieces  of  equipment,  for  example  computer  hardware  and 
software,  but  also  human  beings.  The  human  element  is  significant  and  cannot  be 
overlooked.  Testing  involves  such  things  as  how  well  people  communicate  and  how 
others  understand  and  interpret  communications  under  extreme  stress.  Unlike  an 
aeroplane  or  a  truck,  test  subjects  cannot  readily  be  taken  into  the  field  and  be 
subjected  to  repeated  tests  using  a  range  of  likely  scenarios  over  a  period  of  time.  Yet, 
in  spite  of  this,  systems  such  as  C2,  public  sector  welfare  programmes  and 
organisations  need  to  be  tested  for  effectiveness  and  performance. 

The  social  sciences  deal  with  both  qualitative  and  quantitative  data  as  a  matter  of 
course.  Their  methods  of  dealing  with  the  analysis  of  qualitative  data  offer  means  for 
the  handling  of  the  evaluation  of  complex  systems  involving  human  participation. 
This  report  will  examine  the  nature  of  the  data  used  by  the  social  sciences,  especially 
qualitative  and  subjective  data.  The  distinction  between  such  data  and  that  used  by  the 
physical  sciences  will  be  discussed  as  well  as  the  notion  that  there  is  no  primacy  of  data 
between  quantitative  and  qualitative  data  per  se.  A  general  overview  of  the  methods 
used  by  the  social  sciences  will  be  presented  as  will  an  example  of  a  field  where  such 
an  approach  has  already  proved  successful.  Finally,  a  minor  case  study  where  this 
approach  has  been  attempted  for  a  C2  project  will  be  briefly  discussed. 

2.  The  Social  Sciences 


T&E  practitioners  are  more  likely  to  be  familiar  with  the  physical  sciences  than  with 
the  social  sciences.  The  physical  sciences  are  '...the  ongoing  refined  and  systematic 
efforts  to  understand  the  inorganic  world  and  the  results  of  those  efforts.  These  are 
generally  divided  into  four  broad  areas;  physics,  chemistry,  astronomy,  and  the  earth 
sciences'(Britannica,  1999).  These  disciplines  deal  almost  entirely  with  physical 
entities,  that  is  items  that  possess  shape,  colour,  temperature,  etc.  The  term  'Natural 
Sciences'  is  sometimes  used  as  well  as  the  term  'physical  sciences'  to  describe  '...any  of 
the  sciences  (such  as  physics,  chemistry,  or  biology)  that  deal  with  matter,  energy,  and 
their  interrelations  and  transformations  or  with  objectively  measurable  phenomena' 
(Webster,  2000). 

The  fundamental  ideas  of  the  social  sciences  have  their  roots  with  the  ancient  Greeks 
and  '...their  rationalist  inquiries  into  the  nature  of  man,  state,  and  morality' 
(Britannica,  1999).  However,  the  social  sciences  in  the  form  of  the  disciplines  by  which 
we  know  them  today  came  about  in  the  mid-19th  century.  Modem  social  sciences  deal 
with  the  social  and  cultural  aspects  of  human  behaviour  and  include  disciplines  such 
as:  anthropology;  sociology;  psychology;  economics;  education,  and  political  science. 
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Law  is  sometimes  mooted  as  a  social  science  while  history  is  considered  to  be 
somewhere  on  the  margin  between  the  social  sciences  and  the  humanities.  The  term 
'behavioural  sciences'  is  often  applied  to  the  disciplines  considered  to  be  social  sciences 
in  an  attempt  to  align  them  with  disciplines  such  as  physical  anthropology  and 
physiological  psychology.  Ehrlich  (2000,  p  2)  notes  the  difference  between  the  social 
and  physical  (natural)  sciences  as  being  '...  modem  natural  science  employs  an 
ontology  that  is  restricted  in  a  particular  way:  it  deals  almost  exclusively  with  solely 
physical  things'.  As  a  result  of  this  division,  workers  in  the  physical  or  natural  sciences 
can  generally  rely  on  being  able  to  obtain  quantitative  data.  The  social  or  behavioural 
sciences  on  the  other  hand  are  dealing  with  human  behavioural  characteristics  such  as 
consciousness  and  feelings  etc.  that  are  not  always  amenable  to  quantitative 
measurement.  While  quantitative  techniques  may  be  useful  in  some  aspects  of  analysis 
in  the  social  sciences,  such  as  with  language,  they  are  not  appropriate  where  the 
analysis  builds  on  intuition  and  on  insights  gained  '...through  deep  immersion  in  and 
dwelling  with  the  data'  (Tesch,  1990,  p  60).  In  these  instances,  the  social  or  behavioural 
sciences  must  rely  on  qualitative  techniques  for  the  obtaining  and  analysis  of  data. 

3.  Distinctions  Between  Data  Categories 

The  distinction  between  quantitative  and  qualitative,  and  subjective  and  objective  data 
needs  to  be  clarified  as  the  terms  are  often  confused.  Subjective  data  is  dependent  on 
the  judgement  of  a  human  being  while  objective  data  is  independent  of  human  opinion 
or  judgement.  For  example,  saying  that  'It  is  very  hot  today'  is  subjective  while  saying 
that  'The  thermometer  reading  today  is  41C'  is  objective  even  though  the  observations 
are  both  in  agreement.  Preece  et  al.  (1994,  p.  719)  provide  the  generally  accepted 
distinction  between  qualitative  and  quantitative  when  they  define  quantitative  data  as 
'...data  that  are  comprised  of  numeric  values'  and  qualitative  data  as  data  '...that  can 
be  categorised  in  some  way  but  which  cannot  be  reduced  to  numerical  measurements'. 
Quantitative  data  deals  with  numbers  but  qualitative  data  deals  with  meanings  that  are 
mediated  mainly  through  language  and  action.  Tesch  (1990,  p  56)  defines  qualitative 
data  as  words  and  qualitative  research  as  research  using  words  predominantly  or 
exclusively  as  data.  Dey(1993,  p  11)  is  less  restrictive  and  defines  qualitative  data  as 
'...virtually  any  kind  of  data:  sounds,  pictures,  videos,  music,  songs,  prose,  poetry,  or 
whatever'. 

Table  1  illustrates  the  various  categories  with  examples  of  items  from  each  category. 
Objective-quantitative  data,  such  as  'airspeed'  requires  the  collection  of  'hard'  data 
such  as  can  be  measured  with  some  form  of  apparatus.  On  the  diagonally  opposite 
comer,  subjective-  qualitative  data  are  very  much  subject  to  human  opinion  eg.  'Can 
the  aircraft  be  landed  on  an  austere  airfield'.  There  is  only  one  real  answer  to  this:  -  it 
can  or  it  cannot. 
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Table  1:  Classification  of  data  (from  AFOTECH  99-101,1995). 


Objective 

Quantitative 

Qualitative 

Objective-quantitative 

Examples: 

•  Average  airspeed 

•  Maximum  detection  range. 

Ob  j  ective-qual  itati  ve 

Example: 

•  Demonstrated  communications 
connectivity.  EMP/HEMP 
survivability. 

Subjective 

Subjective-  quantitative 

Examples: 

•  Rating  of  workload  and 
fatigue. 

•  Average  number  of  crew 
errors. 

Subjective-qualitative 

Examples: 

•  Test  team  judgment  on  capability 
to  land  at  austere  airfield. 

•  Test  team  judgment  on  readability 
of  a  manual. 

4.  Value  of  Qualitative  Data 


Sometimes  the  opinion  is  expressed  that  quantitative  data  takes  precedence  over 
qualitative  data.  The  sub-title  for  the  paper,  from  which  this  report  has  been  adapted, 
is  'Real  men  don't  collect  soft  data'.  It  is  from  a  book  of  the  same  name  by  Gheradi  et 
al.  (1987)  who  were  alluding  to  this  notion  of  one  type  of  data  being  inferior  to  the 
other.  Instances  of  this  idea  can  also  be  seen  in  Deming  (1982,  p.  68)  who  notes  that 
terms  such  as  round,  strong,  safe,  etc.  have  no  meaning  for  use  in  '...business  or  in 
government'  unless  they  are  able  to  be  defined  in  statistical  terms.  Preece  et  al.  (1994, 
p.  517)  when  discussing  measures  (metrics)  for  software  states  that  'Once  something 
can  be  measured  numerically,  you  move  away  from  the  world  of  opinion  and 
intuition'.  They  quote  Lord  Kelvin  who  said  in  an  address  to  the  Institution  of  Civil 
Engineers  in  1883; 


I  often  say  that  when  you  can  measure  ivhat  you  are  speaking  about,  and 
express  it  in  numbers,  you  know  something  about  it;  but  when  you  cannot 
express  it  in  numbers,  your  knowledge  is  of  a  meagre  and  unsatisfactory 
kind' 


Roche  et  al.  (1991,  p.  165)  dispute  this  and  state  that  it  is  a  shortsighted  view.  They 
argue  that '. .  .the  most  obvious  or  readily  quantifiable  measures  may  not  necessarily  be 
the  right  ones  at  all'  (p.  169).  A  study  of  the  Combined  Bomber  Offensive  (CBO) 
against  Germany  by  the  British  and  US  Air  Forces  in  WWII  was  used  to  illustrate  that, 
at  a  strategic  level,  there  are  '...considerations  which  go  well  beyond  anything  we  can 
quantify'  (Roche  et  al.  1991,  p.  184).  Dey  (1993,  p  13)  states  that  'It  would  be  wrong  to 
assume  that  quantitative  data  must  take  precedence  over  qualitative  data  simply 
because  it  uses  numbers'.  Fie  uses  the  example  where  body  weight  may  be  established 
quantitatively  but  where,  for  a  weight  watcher,  qualitative  judgements  on  whether 
they  are  'thin',  'fat',  or  'just  right'  might  be  the  judgements  that  count  in  a  social 
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context.  Again,  paint  may  be  quantified  in  terms  of  lightness,  saturation,  and  hue  but 
the  judgement  that  counts  could  be  the  qualitative  one  of  whether  it  'clashes'  with 
other  colours  or  is  an  appealing  colour.  The  quantitative  measurements  of  colour  may 
be  meaningful  for  a  paint  technologist  but  qualitative  measurements  are  of  more  use  to 
the  vast  majority  of  the  population. 

During  the  formative  period  of  modem  social  sciences  in  the  19th  century,  the  success 
of  the  physical  sciences  in  this  period  was  noted  and  the  social  sciences  tried  to  emulate 
this  success  '...through  the  adoption  of  quantitative  techniques'  (Dey,  1993,  p.3).  There 
is  a  heavy  emphasis  still  on  quantitative  techniques  in  the  social  sciences  but  there  is 
also  a  growing  recognition  that  qualitative  techniques  have  a  significant  role.  Miles  et 
al.  (1994,  p.  40)  recognises  that  there  is  a  place  for  both  quantitative  and  qualitative 
data  when  they  say  '. .  .we  have  to  face  the  fact  that  numbers  and  words  are  both  needed 
if  we  are  to  understand  the  world'.  Rossi  et  al.(1993,  p.  437)  when  discussing  the  merits 
of  a  quantitative  as  against  a  qualitative  approach,  express  a  similar  view.  They  note 
that  '...each  approach  has  utility,  and  the  choice  of  approaches  depends  on  the 
evaluation  question  at  hand'.  They  stress  that  the  qualitative  approach  can  '...play 
critical  roles  in  program  design  and  are  important  means  of  monitoring  programs'. 
Posavac  et  al.  (1992,  p.  211)  note  how  quantitative  techniques,  while  suitable  for 
evaluation  when  the  goals  are  clear,  are  not  suitable  when  the  goals  are  vague  ie.  when 
different  stakeholder  groups  were  able  to  read  their  own  goals  into  the  objectives  of  the 
enterprise.  They  note  the  advantage  of  qualitative  techniques  when: 

•  total  objective  measurement  is  '. .  .extraordinarily  difficult,  if  not  theoretically 
impossible'; 

•  it  cannot  be  assumed  that  goals  will  be  clearly  stated;  and 

•  programmes  are  highly  sensitive  to  context  or  scenario. 

That  research  can  be  done  by  talking  to  people  and  observing  and  by  trying  to  make 
sense  of  what  they  observe  should  really  not  be  surprising  to  systems  engineers.  In  the 
world,  everything  is  eventually  linked  to  everything  else  and  from  this  comes 
emergent  qualities  as  well  as  the  need  to  take  a  holistic  view  of  the  world.  Instead  of 
arguing  for  the  primacy  of  one  type  of  data  over  the  other,  there  seems  to  be  merit  in 
recognising  that  both  types  are  able  to  make  a  contribution. 

5.  Social  Science  Methods  of  Data  Analysis 

The  approaches  taken  by  the  social  sciences  to  analyse  qualitative  data  have  evolved 
only  in  relatively  recent  times  and  there  is  a  wide  range  of  names  given  to  the  methods 
used  and,  for  the  uninitiated  at  least,  a  seemingly  incomprehensible  associated  jargon. 
Tesch  (1990,  p  58)  lists  some  46  different  methods  in  common  use  which  she  then 
breaks  up  into  four  'cognitive  maps'  that  loosely  describe  the  purpose  of  the  research 
methods.  These  are: 

•  the  study  of  the  characteristics  of  language; 

•  the  discovery  of  regularities; 
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•  seeking  to  discern  meaning;  and 

•  methods  based  on  reflection. 

Miles  et  al.  (1994,  p.  5)  describes  Tesch's  (1990)  taxonomy  as  one  of  establishing 
purpose  and  proceeds  to  describe  other  taxonomies  using  methods  employed  to  obtain 
data  and  even  traditions  of  qualitative  research.  Dey  (1993,  p.  5)  considers  that  the 
plethora  of  taxonomies  used  by  social  scientists  is  really  defensive  posturing  to  either 
emphasise  or  even  exaggerate  '..  .the  subtleties  and  complexities  involved  in  qualitative 
analysis'.  He  points  to  the  need  to  explain  how  analytic  principles  and  procedures  are 
applied  in  social  science  instead  of  emphasising  how  different  methods  are  applied  to 
particular  aspects  of  research.  He  considers  that  in  all  these  methods  there  is  a 
'...common  emphasis  on  how  to  categorise  data  and  make  connections  between 
categories.  These  tasks  constitute  the  core  of  qualitative  analysis'. 

Tesch  (1990)  shows  that  one  purpose  of  qualitative  research  can  be  to  discover 
regularities  and  this  does  have  relevance  in  the  evaluation  of  complex  systems  such  as 
Cl.  The  cognitive  map,  shown  as  Figure  1,  demonstrates  that  this  purpose  can  be 
further  divided  into  two  sub-purposes,  of  which  the  one  of  particular  interest  for  T&E 
is  that  of  'discerning  of  patterns  as  deficiencies,  ideologies'.  'This  is  shorthand  for 
saying  that  these  types  of  research  are  not  used  for  the  development  of  theoretical 
notions,  but  for  practical  scrutiny  of  human  situations,  and  often  also  for  the  formation 
of  alternative  solutions  where  problems  are  found  to  exist'  (Tesch,  1990,  p  65).  This 
ability  to  undertake  'practical  scrutiny  of  human  situations',  and  the  'formation  of 
alternative  solutions'  makes  qualitative  analysis  a  useful  means  of  undertaking  T&E  in 
socio-  technical  systems.  Figure  1  also  shows  that  methods,  such  as  'action  research'  or 
'qualitative  evaluation',  may  be  used  to  achieve  the  stated  purpose. 


the  discovery  of  regularities 


identification  (and  categorisation) 
of  elements,  and  exploration  of 
their  connections 


ethnographic 

content 

analysis 


transcendental 

realism 


ecological 

psychology 


event  structure 
analysis 


grounded 

theory 


discerning  of  patterns 


as  deficiencies, 
ideologies 


as  culture 


in  conceptualisation 


phenomenography 


holistic 

ethnography 


5  socialisation 


qualitative  evaluation, 
action  research, 
collaborative  research, 
critical/emancipatory  research 


educational 
ethnography, 
naturalistic  inquiry 


Figure  1:  Mapping  of  qualitative  analysis  purposes.  After  Tesch  1990 
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The  methods  listed  for  the  purpose  of  discerning  deficiencies  have  various  attributes 
that  make  them  suitable  as  methods  for  the  T&E  of  socio-technical  systems  where 
qualitative  data  either  is  all  that  is  available  or  predominates.  Brief  descriptions  of  the 
methods  are: 

•  evaluation  research  '...seeks  to  determine  whether  an  action  has  the  results  it  is 
meant  to  have'  (Tesch,  1990,  p  65)  and  the  most  common  'actions'  are  the 
programmes  implemented  by  public  service  agencies  using  experimental  design. 
This  research  fits  into  the  concept  of  establishing  T&E  programmes  based  on 
Measures  of  Effectiveness  (MOE)  and  Measures  of  Performance  (MOP)  (Sproles, 
2000a); 

•  action  research  is  less  passive  than  evaluation  research  and  is  geared  towards 
improvement  of  unsatisfactory  situations.  It  encompasses  the  human  activity 
systems  methodology  of  Checkland  (1981)  and  can  '...be  used  as  a  research  tool  for 
investigative  or  pilot  research,  and  generally  for  diagnosis  or  evaluation.'  (Dick, 
1997).  As  such,  it  is  more  appropriate  for  formative  or  developmental  T&E. 
Collaborative  enquiry  and  critical/emancipatory  research  are  forms  of  action 
research. 


6.  A  Legitimate  Approach 

The  ability  to  use  qualitative  methods  or  approaches  to  establish  regularity  indicates 
that  the  use  of  such  methods  is  scientific  and  legitimate.  Wilson(1999)  states  that  'All 
science  is  the  search  for  unity  in  hidden  likenesses.  The  scientist  looks  for  order  in  the 
appearances  of  nature  by  exploring  such  likenesses'.  Britannica  (1999)  puts  the  case 
that  science  is  the  knowledge  of  '...natural  regularities  that  is  subject  to  some  degree  of 
skeptical  rigour  and  explained  by  natural  causes'.  House  (1980,  p.279)  states  that 
'Naturalistic  generalisation  employs  a  special  kind  of  qualitative  argument'. 
Naturalistic  can  be  defined  as  'The  scientific  observation  of  events  as  they  occur 
without  trying  to  manipulate  them  in  the  form  of  an  experiment  in  any  other  way' 
(Statt,  1993,  p.87).  House  (1980,  p.  280)  discusses  how  evaluation  based  on  naturalistic 
observation  is  focussed  on  the  understanding  of  interactions,  and  searches  for 
explanation  rather  than  prediction. 

7.  Qualitative  Approach  in  Human  Service  Programmes 

The  use  of  social  science  methods  of  qualitative  analysis  to  establish  effectiveness  has 
already  been  successfully  applied  to  human  service  programmes.  These  include  areas 
such  as  the  criminal  justice  system,  education,  training  and  safety  programmes  for 
industry  and  business,  and  in  many  sections  of  public  administration.  There  is  a  body 
of  literature  that  discusses  how  these  social  science  methods  can  be  applied  to  what 
Posavac  et  al  (1992,  p  2)  terms  'Programme  Evaluation'  which  is  '...a  new  and  exciting 
applied  social  science'.  While  Posavac  et  al  (1992)  describes  many  of  the  quantitative 
techniques  used  in  the  social  sciences,  chapter  12  is  devoted  to  the  analysis  of 


7 


DSTO-GD-0263 


qualitative  data.  In  this  chapter  they  acknowledge  that  evaluation  cannot  always  place 
an  emphasis  on  experimental  control  providing  quantitative  data.  They  also  establish 
that  qualitative  research  can  be  rigorous  in  spite  of  the  element  of  subjective  judgement 
that  it  often  includes. 

8.  The  Battlefield  Command  Support  System  - 

A  Case  Study 

In  early  1999,  the  Command,  Control,  Communications,  and  Intelligence  Operations 
Analysis  Group  (C3IOA)  of  the  Defence  Science  and  Technology  Organisation  (DSTO) 
was  asked  to  evaluate  the  effectiveness  of  the  Australian  Army's  Battlefield  Command 
Support  System  (BCSS).  The  difficulty  in  carrying  out  such  a  task  is  the  problem  of 
establishing  if  the  C2  system  has  or  has  not  made  a  contribution  to  the  accomplishment 
of  the  operational  mission.  Whatever  contribution  BCSS  may  make  is  generally 
masked  by  the  contribution  made  by  other  component  systems.  Because  these 
contributions  are  combined  to  produce  an  emergent  property  of  the  total  system,  one 
can  rarely  be  separated  from  the  others.  It  is  only  when  some  gross  contribution  is 
made,  for  better  or  for  worse,  that  an  observer  can  make  such  a  distinction.  It  can  be 
shown  that  the  MOEs  for  systems  such  as  BCSS  lie  in  the  contribution  that  they  make 
to  the  accomplishment  of  the  military  commander's  mission  (Sproles,  2000b).  This  is 
usually  very  difficult  to  establish  by  experiment  as  the  contribution  made  is  generally 
lost  among  the  synergy  that  produces  the  emergent  properties  of  the  meta-system 
represented  by  the  military  force  itself. 

A  complication  that  arose  with  this  study  was  that  there  was  no  opportunity  to  study 
BCSS  in  the  field  nor  was  there  any  opportunity  to  carry  out  repetitive  testing  to 
establish  regularities.  Situations  such  as  this  are  common  not  only  in  Australia  but 
overseas  (Levi,  pers.comm.,  1998)  necessitating  the  T&E  organisations  to  seek  other 
means  to  provide  the  test  and  evaluation  of  systems.  In  this  instance,  it  proved 
possible  to  use  methods  from  the  behavioural  sciences  to  accomplish  the  task. 

The  approach  taken  in  this  case  was  much  along  the  lines  suggested  by  Dey  (1993)  in 
Chapter  3.  A  snapshot  was  taken  of  an  Army  brigade  that  had  experience  in  the  field 
with  the  use  of  BCSS.  The  process,  as  described  by  Dey  (1993),  was  to  collect  data,  sort 
it,  categorise  it,  and  to  make  connections  enabling  a  conclusion.  The  data  collection 
was  achieved  by  interviewing  a  cross  section  of  members  of  the  formation  from  the 
commander  down  to  private  soldiers.  Interviews  were  carried  out  face-to-face  with  the 
interviewees  at  their  workplace.  The  questions  were  prepared  beforehand  and  were 
categorised  based  on  existing  knowledge  of  how  an  effective  C2  system  was  likely  to 
contribute  to  mission  effectiveness.  On  completion  of  the  interviews  the  data  was 
entered  into  Nvivo,  a  computer  based  qualitative  analysis  tool.  This  permitted  the 
responses  to  be  categorised  either  into  the  existing  categories  or  for  new  categories  to 
be  established.  At  all  times,  the  aim  was  to  funnel  data  into  relevant  categories  for 
analysis  to  enable  a  conclusion  to  be  made  on  the  contribution  of  BCSS  to  mission 
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effectiveness.  In  the  process,  some  existing  categories  were  amended  in  scope,  new 
categories  were  established,  and  some  categories  were  allowed  to  overlap.  When  this 
was  completed,  it  was  possible  to  make  comparisons  between  the  responses  and  to 
make  connections  between  the  different  pieces  of  data.  A  search  was  made  to  establish 
regularities,  exceptions  to  these  regularities,  and  even  singularities  ie.  when  comments 
were  made  that  no  one  else  had  made. 

It  was  possible  from  this  to  develop  a  picture  of  how  effective  BCSS  was  at  this  stage 
and  to  draw  some  conclusions  on  its  future  effectiveness.  Admittedly  this  was  not  a 
rigorous  process  due  to  the  limited  opportunity  and  the  taking  of  samples  at  one  point 
of  time  only.  On  the  other  hand,  the  data  is  still  available  for  future  use.  Given 
opportunities  in  the  future  to  carry  out  a  similar  exercise,  the  results  can  be  compared 
to  build  up  a  richer  picture.  The  existing  data  can  be  combined  with  the  new  data  and 
even  re-categorised  if  so  desired.  This  exercise  demonstrated  that  qualitative 
techniques  were  feasible  in  establishing  the  effectiveness  of  a  socio-technical  system. 

9.  Conclusion 

The  use  of  qualitative  data  and  qualitative  analysis  is  a  legitimate  tool  to  establish 
regularities  in  a  system.  In  socio-technical  systems,  such  as  military  C2,  they  may  offer 
the  only  feasible  means  to  establish  the  effectiveness  of  a  system.  The  techniques  used 
to  gather  and  analyse  the  data  have  been  developed  over  a  reasonable  length  of  time 
by  social  science  practitioners  and  have  been  shown  to  be  able  to  produce  rigorous 
results.  When  such  techniques  were  attempted  by  DSTO,  albeit  in  a  limited  way,  they 
produced  believable  results.  There  seem  to  be  reasonable  grounds  to  carry  out  further 
research  in  this  area  to  enable  these  techniques  to  be  adapted  for  the  T&E  of  socio- 
technical  systems  that  are  proving  resistant  to  T&E  by  techniques  grounded  in  the 
physical  or  natural  sciences. 
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